Visual localization method

ABSTRACT

There is provided a visual localization method comprising: (a) transmitting data representative of one or more detected visual features from a mobile device to a server; (b) estimating the location of the mobile device at the server based on the visual features received from the mobile device; (c) transmitting reference data associated with the estimated location from the server to the mobile device; and (d) the mobile device determining its location based on the reference data received from the server.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.13/816,093, filed on Jul. 9, 2013, now allowed, which is a NationalStage filing under 35 U.S.C. § 371 of PCT Application No.PCT/EP2011/057901, filed May 16, 2011, which claims priority to EuropeanPatent Application No. 10172403.7, filed Aug. 10, 2010, the contents ofwhich are incorporated by reference herein.

BACKGROUND

Information about the location, orientation, and context of a mobiledevice is of central importance for future multimedia applications andlocation-based services (LBS). With the widespread adoption of moderncamera phones, including powerful processors, inertial measurementunits, compass, and assisted GPS receivers, the variety of location andcontext-based services has significantly increased over the last years.These include, for instance, the search for points of interest in thevicinity, geo tagging and retrieval of user generated media, targetedadvertising, navigation systems, social applications etc.

While satellite navigation systems can provide sufficient positioningaccuracy, a clear view to at least four satellites is required, limitingits applicability to outdoor scenarios with few obstacles.Unfortunately, most interesting LBS could be provided in denselypopulated environments, which include urban canyons and indoorscenarios. Problems may be caused by multipath effects, which are evenmore severe if the user is traveling on the sidewalks and not in themiddle of the street.

As GPS is virtually not available in indoor environments and thelocalization accuracy in urban canyons is insufficient, alternativepositioning mechanisms, which can complement the available systems, arerequired.

One approach may consist in using images recorded on the mobile deviceas a visual fingerprint of the environment and matching them to anexisting georeferenced database like Google Street View or MicrosoftStreet-Side views. In contrast to WiFi based indoor localizationsystems, no infrastructure, which grows in complexity with the size ofthe environment, is required. Further, LBS do not only rely on a preciselocation and orientation information to determine the user's actualfield of view but also benefit from information on its content likeexhibits, store names, trademarks, etc., which can be derived from theimages the user is intentionally recording. Ideally, the poseinformation from visual localization is fused with all other availablesensor data providing location or orientation like GPS, IMU, WiFi, orCell-IDs if available.

The main challenge for visual localization is to rapidly and accuratelysearch for images related to the current recording in a largegeoreferenced database. This task is known as Content Based ImageRetrieval (CBIR). Objects, recorded at different size, pose, and withvarying background have to be distinctively described and efficientlyretrieved from a database. The application of CBIR to locationrecognition complicates these requirements.

In particular, images captured with a mobile device are used to retrievethe spatially closest image from a georeferenced dataset. This could,for instance, include the 360° panoramic images from Google Street View,which can be fetched from the web. Typically, only sparse reference datacan be assumed. For instance, Street View panoramas are available onlinewith varying inter-panorama distances, typically in the range of 12 to17 m. However, there are problems associated with wide baselines.Whereas distant buildings can be well associated among the views, closeobjects like the train station or the tree are difficult to match evenfor a human observer. The description of distinct objects is complicateddue to the three-dimensional structure of the environment and theresulting occlusions and overlaps. Further, different lightingconditions between the query and database image, which cause shadows andreflections, can change the visual appearance of the scene. Also, bothquery and database images typically contain dynamic objects, like carsor pedestrians, which lead to significant differences between matchingviews. As advertisements or even buildings alter over time and seasonschange the appearance dramatically, a dynamic update process for thedatabase is required. Due to the properties of mobile device cameras,query images are typically affected by motion blur and provide a limitedfield of view, which makes it difficult to match them against highresolution panoramas. Additionally, limitations on the processing power,battery capacity, and network performance require low complexityapproaches on the mobile device and efficient communication includingdata compression.

Finally, very low retrieval times are an essential prerequisite for mostLBS due to the rapidly changing field of view of the mobile devicecaused by user motion and constantly changing user attention.

Whilst there are different known image retrieval algorithms, the majorbottleneck is the communication delay introduced by feature uploading.Including network delay, communication timeouts, and the retrievalitself, the delay until the client receives results from the server maybe insufficient for some location-based services due to user motion anddynamically changing user attention.

Accordingly, in one embodiment the present invention aims to address twocentral challenges of mobile visual location recognition, namely thecomplex retrieval task and the communication delay.

Moreover, in order to achieve the required low query time, tree-basedbag-of-feature (BOF) approaches are typically used, which quantize imagedescriptors into visual words.

The retrieval of images or image sequences in large databases has beenstudied extensively during the last decades. Object retrieval andlocation recognition are among the most known applications in thisfield. While image retrieval results can be efficiently improved viaBayesian filtering in location recognition scenarios, the requirementson the query time are very stringent. A typical example would be anonline service providing location information based on image recordingsfrom mobile devices and a geo-tagged reference image database likeGoogle Street View. In this scenario, images are typicallydimensionality reduced on the mobile device with the aid of robustfeatures like SIFT or SURF. The extracted features are sent to a server,which has to compute the position estimate within a few milliseconds tomeet the stringent real-time requirements of mobile locationrecognition. The ability to rapidly estimate the absolute location isessential to continuously limit the temporally increasing uncertainty ofthe user's position and thus the computational complexity.

In feature based retrieval approaches, the similarity of images istypically determined by a score based on the count of matching highdimensional feature descriptors. To avoid a query time, which scaleslinearly with the number of database images, efficient indexingstructures like the popular kd-tree are typically used. These treesperform an approximate k-nearest neighbor search to achieve query timeslower than the one obtained for a linear search for dimensions higherthan 10. However, backtracking through neighboring leaves, which isrequired to achieve reasonable retrieval accuracy, amounts for asignificant percentage of the overall query time. Further, in theseapproaches the descriptors of every image have to be stored, whichresults in a linear increase of the database size.

Accordingly, in another embodiment the present invention aims to addressthe challenge of rapid location recognition, and in particular toprovide a location recognition system that reduces data processing timeand expenses as well as data storage requirements.

SUMMARY OF THE INVENTION

The present invention is recited in the independent claims. Preferred,non-limiting embodiments are recited in the sub-claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an overview of a basic visual location system.

FIG. 2 is an illustration of a tree-based BoF quantization and indexingstructure. Image patches assigned to a single node decrease in diversitywith increasing descriptor quantization. The leaf nodes are defined asvisual words with associated inverted files. These reference the imagesto be scored when a query image patch is quantized to the respectiveword.

FIG. 3 is an Overview of a visual location recognition system where theretrieval task is shifted from the server to client by transferring thequantizer and compressed inverted file system of the most suitablesubregion. Dashed lines indicate the data transmission duringinitialization. Once the transfer of quantizer and inverted file isfinished no further network delay is introduced.

FIG. 4 is an overview of a visual location recognition system whererelevant parts of the vocabulary are periodically pushed to the client.The selection of the transmitted visual words is based on the locationretrieval performed at the server using the query features uploaded fromthe client. The pose estimation is performed on the mobile deviceutilizing these preloaded adaptive partial vocabularies. Blue linesindicate that the network delay does not influence the time required forthe location recognition.

FIGS. 5 to 9 relate to embodiments of the present invention including anovel approach for rapid mobile location recognition which is based onwhat is referred to as Multiple Hypothesis Vocabulary Tree (MHVT).

DESCRIPTION OF EMBODIMENTS OF THE PRESENT INVENTION

Bag-of-Features Approach

FIG. 1 illustrates a basic architecture of a visual location recognitionsystem which is in principle similar to that of a mobile visual productsearch system, with the difference that not a single image but acontinuous sequence of images is used to query a reference database. Asa first step, robust local features are extracted from the individualvideo frames to distinctively describe their content. Once the featuresare extracted, they are sent to the server where content based imageretrieval algorithms (CBIR) are employed to identify the visually mostsimilar reference image in the database.

One approach to this is the Bag-of-Feature (BoF) approach as illustratedin FIG. 2. The BoF approach reformulates the image retrieval probleminto a text retrieval problem by quantizing the high-dimensional featuredescriptors into so called visual words with the aid of the k-meansalgorithm. At fine quantization, descriptors associated to a word followa texture pattern that is sufficiently represented by their mean. Thus,an image is no longer represented by its descriptors but by a visualword frequency histogram, the so called Bag-of-Features (BoF) vector.Similarity between two images is measured by the distance between theirBoF vectors, which can be efficiently computed using inverted files. Themajor advantages of BoF based image retrieval are its compactness, thereduced storage requirements, and the low query times.

Feature Extraction and Selection

It has been found that detectors arranged to find distinctive imagepatches based on the texture produce a high number of features belongingto plants (e.g. foliage), which are unreliable due to the constantlychanging texture and seasonal changes. According to an embodiment of theinvention, retrieval performance is improved by excluding those featuresbased on the color distribution of the corresponding image patch, whichis mostly green. Hence, not only the quantization, indexing, andverification but also the selection of features is a major issue invisual location recognition. The selection of the most relevant featuresincreases the performance and reduces the amount of data to betransmitted. Further, rotation invariance of the descriptor is notrequired as it can be assumed that database as well as query images arerecorded upright or rotated by 90°. This not only significantly reducesthe extraction time on the mobile device but also increases thedistinctiveness of the descriptors.

Adaptive Partial Vocabularies

It has been found that the performance of a BoF-based mobile locationrecognition system can be improved based on the above mentioned featureselection and by making use of prior knowledge of the user's location.For example, vague prior knowledge on the location of a mobile devicecan be derived from cell-IDs. This can be exploited to increase theprecision, reduce complexity and limit the requirements on thescalability of the adopted location recognition approach.

Furthermore, by transferring only selected features from the client tothe server, the overall latency can be reduced. Still, this may notsufficient for some visual location recognition applications. Ideally, auser is notified about a LBS in the very moment the camera records thecorresponding store, exhibit, trademark, etc. Due to the rapidlychanging field of view and dynamically changing user attention, very lowsystem latency is essential for LBS to be perceived as useful. With thebasic system architecture shown in FIG. 1 the response time alwayssuffers from the round trip time, which ranges between 0.3 and 0.9seconds on 3G networks, and timeouts in addition to the time required toactually transfer the data. The architecture almost exclusively uses theslower uplink channel for uploading query features, whereas the downlinkis usually more than 5 times faster. Further, the selection of featuresto be transferred can hardly be based on database statistics, as thiswould require the quantization into visual words.

A possible approach to solve this problem is to transfer the quantizeras well as the inverted file to the client and perform the locationretrieval exclusively on the mobile device as shown in FIG. 3. With theapproach described above to integrate prior knowledge, the size of theinverted file system may be reduced. In addition, suitable compressionmay be applied. Nevertheless, although this approach eliminates thenetwork delay by performing the location retrieval step on the mobiledevice, the amount of data to be transmitted via the downlink may resultin a time consuming initialization of the system. During this phase,localization may need to be performed according to FIG. 1, wherefeatures are sent to the server and the position can be updatedperiodically.

According to a preferred embodiment of the present invention, based onthe features transmitted from the mobile device to the server, thelocation of the client is periodically estimated at the server. Thisallows for a high performance with respect to precision and recall. Withthese periodic location estimates (e.g. every 3 seconds) only thereference data within a certain vicinity at the mobile device is neededfor a local refinement of the location estimate. This is usually only afraction of the full vocabulary and inverted file system.

If F={f₁, f₂, . . . , f_(N)} is considered to be the set of features ofone query frame and V={v₁, v₂, . . . , v_(L)} to be the set of visualwords (i.e., the full vocabulary), the quantization functionq_(V)(f_(i))=v_(j) assigns each feature f₁ to a visual word v_(j) in thefull vocabulary V. Hence, Q(F|V)={v=q_(V)(f)|fϵF}=V_(F) ⊂V determinesthe subset of visual words which represents a particular video frame. Ifinstead of the full vocabulary only a random partial set of visual wordsV_(R) ⊂V is used the feature quantization result will be different inmost cases. However, using the subset of visual words representing theframe itself V_(F)=Q(F|V) as the partial vocabulary would not change theresult of the quantization as shown in Eq. 1.Q(F|V _(F))=Q(F|V)  (1)

Hence, only this part of the full vocabulary needs to be available atthe client to obtain the same results as if using the full vocabulary.However, this equation only holds for a specific set of features F and apartial vocabulary V_(F) would need to be sent to the client for eachframe. Ideally, a partial vocabulary that includes the correct visualwords is identified to process multiple consecutive frames without theneed to know their features apriori. Since V_(F) can be extended byother subsets of the full vocabulary S⊂V without changing thequantization result, as shown in Eq. 2, partial vocabularies at theclient that have a sufficiently high probability of including V_(F) canbe used.Q(F|V _(F) ∪S)=Q(F|V)  (2)

To limit the amount of data to be transferred to the client, thesmallest partial vocabulary that includes the unknown V_(F) with highprobability is identified.

Based on the periodic location estimates obtained at the server, thereis prior knowledge on the location in form of the top K retrievedlocations. Thus, the partial vocabulary can be based on the visual wordsthat are part of panoramas located at these candidate locations.Features of frames recorded at these location are expected to bequantized to the corresponding visual words. Hence, the probability thatV_(F) of these frames is part of this partial vocabulary is very high.

Hence, according to an embodiment it is envisaged to periodically sendonly these relevant visual words together with their associated invertedfiles to the client to allow for a local pose estimation on the mobiledevice within a limited area as shown in FIG. 4. This is achieved bymatching query features on the mobile device to the partial vocabulary.

For example, this can be performed at about 10 fps. At an average motionof 1.2 m/s the visual words of two neighboring panoramas (distanceranging between 12-17 m) this would allow for a navigation of at least10 seconds or 100 frames without an update of the partial vocabulary.Due to imperfect location retrieval at the server and/or degeneratedquery features sent, the probability that the first ranked panorama islocated at the actual location is about 60%. To achieve a probability ofabout 90% the top 5 ranked panoramas need to be considered to build apartial vocabulary.

It has been found that querying the partial vocabulary, which is basedon the top K panoramas, actually results in an increased precisioncompared to the full vocabulary. This is due to the fact that with ahigh probability the correct location is among the top K results, butpossibly not at the beginning of this ranking. As the partial vocabularyconstrains the set of possible location candidates, the quantization ofthe features of a subsequent query frame to this vocabulary can beconsidered as a resorting among the original top K results. The panoramathat fits best to both queries is implicitly ranked highest.

As described above, features that vote for one of the preselectedlocations are matched to the same visual words as if using the fullvocabulary. Only those features that refer to other locations with thefull vocabulary can happen to be matched to a different visual word whenconsidering a partial vocabulary. Applying a threshold on the maximallyaccepted distance between the feature and the visual word results in amarginal further improvement.

Thus, the adaptive vocabulary is based not on all visual words alreadytransferred to the mobile device but only those that provide asufficiently high probability of including the correct location.

To make best use of the available data rate of the downlink, a priorityqueue of locations and their visual words is updated every time a newset of features is sent from the mobile device to the server. Thus,those visual words that belong to the panoramas with the highestprobability of representing the actual location are constantly sent.This probability can be determined based on the score of the retrievalor with the aid of temporal filtering approaches like Bayesian filters.For example, with a 3G downlink, the visual words and associatedinverted file entries of at least 2 panoramas (each comprisingapproximately 1000 visual words) can be transferred per second. Withinthe time a feature set is uploaded we can download the visual words ofabout 6 locations.

As more and more visual words are transferred to the client, the chancesthat visual words in the priority queue are already available at themobile device increase and hence they do not have to be sent again. Toincrease the time the client can navigate based on the partialvocabulary, visual words of panoramas that are adjacent to the top Klocation estimates are added to the priority queue. Since theysignificantly overlap with respect to the visual words, only a limitedamount of additional data has to be transferred. To speed up thequantization of feature descriptors to the adaptive partial vocabulary,which includes at most 50 k visual words if the uncertainty about theactual location is large, randomized kd-trees are sent to the mobiledevice. As they require about 10 KB each (for 50 k visual words), afrequent update can be carried out.

Hyperplanes and Overlapping Buffers

In another embodiment the present invention uses a novel approach forrapid mobile location recognition which is based on what is referred toas Multiple Hypothesis Vocabulary Tree (MHVT). The MHVT maximizes theprobability of assigning matching descriptors to the same visual wordwhile minimizing the query time. This is achieved by introducing anoverlapping buffer around robustly determined separating hyperplanes toallow for a soft quantization and an adaptive clustering approach.

Based on this approach, a high accuracy and performance can be achievedwithout requiring search tree backtracking.

In particular, a quantization structure to visual vocabularies may beemployed that includes overlapping buffers, thereby to decrease thequery time substantially in high dimensional spaces. Databasedescriptors are allowed to follow the hypothetical paths through abinary search tree that a matching query descriptor could traverse.Thus, the probability of quantizing a matching descriptor and the queryto the same leaf of the binary search tree is significantly increasedwhile minimizing the loss in distinctiveness of the visual words. Thisstands in contrast to conventional hierarchical scoring, as the scoringenergy is concentrated to the leaf nodes. Hence, time consuming memoryaccess can be avoided.

Moreover, instead of applying a large branching factor k, intended toimprove quantization in k-means trees, the structure is limited to abinary tree to minimize the query time. The ratio between the requiredquery vector comparisons for a binary tree relative to a tree withbranching factor k is given by Equation (3), where L corresponds to thedepth of the k-ary tree:

$\begin{matrix}{\frac{{Comp}_{binary}}{{Comp}_{k - {ary}}} = {\frac{\log_{2}\left( k^{L} \right)}{k \cdot L} = \frac{\log_{2}(k)}{k}}} & {{Equation}\mspace{14mu}(3)}\end{matrix}$

Accordingly, the performance of the quantization structure issubstantially improved by optimizing the determination of separatinghyperplanes, as described in more detail below.

In a preferred embodiment, the present invention comprises weightingvisual words corresponding to the properties of the query descriptors inan inverted file scheme. In particular, as in vocabulary trees aquantization into corresponding image textures instead of a fixed numberof nearest neighbors has to be performed, a strategy to avoidoverfitting of descriptor clusters is employed. To integrate knowledgeon the importance of query descriptors, a framework is employed thatperforms a weighting of visual words within an inverted file scheme.This allows accounting for the probability of incorrectly quantizing aquery descriptor and effectively matching camera images with panoramaviews, as described below.

In another preferred embodiment, the present invention comprisesproposed spatial database segmentation and a fusing scheme facilitatethe integration of prior location knowledge to increase retrievalprecision. By segmenting the search space into multiple overlappingregions and fusing the results of the corresponding vocabulary trees, anincrease in precision is achieved while adapting the utilized resourcesto the current uncertainty about the position of the user. This conceptfacilitates the integration of prior location information.

More particularly, according to an approach employed in an embodiment ofthe present invention, hyperplanes separate the nodes of the decisiontree into two mutually exclusive child nodes. Overlapping buffers areintroduced in order to prevent a separation of descriptors too close tothe hyperplanes, i.e. descriptors that cannot be clearly distinguished.Such descriptors are stored in overlapping buffers. Thus, descriptors inthe overlapping buffers are not separated from each other at the currentnode. Instead they are assigned to both child nodes.

In particular, the search tree may be based on metric-trees thatorganize d-dimensional data hierarchically by separating the spaceiteratively with hyperplanes. As is illustrated in FIG. 5, at each node,a vector {right arrow over (u)}, spanning between the two mutuallyfarthest points (pivots), and the mean or median of all data pointsprojected onto {right arrow over (u)}, are determined. Thus, {rightarrow over (u)}∥{right arrow over (u)}∥ is the normal vector of thehyperplane that separates the node at the mean or median resulting intwo mutually exclusive child nodes. The process proceeds recursivelyuntil the number of data points is less than the defined maximum, theso-called naive count. This count represents the maximum number ofdescriptors that will be considered for linear search once a queryinvestigates a particular leaf node.

During the decision process, a query descriptor proceeds down the treeto evaluate on which side of the hyperplane it is lying. Descriptorslying close to the splitting boundaries have a high probability ofmatching to a descriptor in the neighboring node and would normallyrequire backtracking to be found. In order to avoid this, an overlappingbuffer around the boundary is introduced. FIG. 5 illustrates thequantization with overlapping buffers. All database descriptorsprojected inside the buffer are assigned to both child nodes as long astheir number does not surpass a certain fraction ρ of the descriptors inthe mother node. Hence, descriptors that cannot be clearly distinguishedby the current decision boundary are not separated from each other atthe current node. The differentiation is delayed to the child nodeswhere the probability of lying far from the separating hyperplane ishigh. As the nodes are no longer mutually exclusive, increased memoryrequirements are entailed which are bounded by the ratio ρ. The maximumtree depth is given in the following equation, where N is the overalldescriptor count and n the naive count:

$\begin{matrix}{{depth}_{\max} = \frac{\log\left( {n/N} \right)}{\log(\rho)}} & {{Equation}\mspace{14mu}(4)}\end{matrix}$

In order to achieve an efficient clustering, the vector u shouldseparate the data points along the direction of largest variation(dashed transverse line in FIG. 5a ). However, as shown in FIG. 5a , thevector spanned by the two mutually farthest points (solid horizontalline) does not necessarily fulfill this task as it is very sensitive tooutliers. This is especially true if the data has already been splitalong multiple axis and the variation is increasingly isotropic.Further, to determine the pivots with linear time requirements,approximations are utilized that again suffer from outliers.

Due to the introduction of overlapping buffers, the performance of thehybrid spill tree may be sensitive to the determination of the directionof maximum variation. This may affect not only the robust assignment ofmatching descriptors to the same child node but also the effectivenessof the overlapping buffers. These effects may be amplified in highdimensional spaces. Hence, a robust, precise but yet simple algorithm todetermine the splitting axis is required to optimize the quantizationperformance may be desirable.

Satisfactory results can be achieved by spanning the vector u betweenthe two farthest point cluster centers with round (√{square root over(n)}·c) descriptors per cluster, where n is the number of descriptors inthe node and c should be in the range of approximately 25%. Thesecentroids are less susceptible to outliers and can be estimated in anapproximate manner. This is done by starting at a random point andfinding its most distant point, which in turn is used to find its round(√{square root over (n)}·c) farthest points. Based on the mean of thiscluster, again the farthest cluster is determined and a mean iscomputed. This process is approximative, similar to the original nodepartitioning method and can be run fast if (√{square root over(n)}·c)<<n using the heap sort algorithm in at most O (n log (√{squareroot over (n)}·c)) steps.

With this improved projection axis u, a better retrieval accuracy can beachieved without backtracking. Further, also with backtracking theperformance is significantly improved. If the time required to build thetree is not relevant and the database is not too large, the optimalsplitting axis can also be determined by the eigenvector correspondingto the largest eigenvalue of the node's covariance matrix.

MHVT

According to an embodiment of the invention the advantageous propertiesof the overlapping buffers are transferred to vocabulary trees,resulting in a novel adaptive clustering strategy to optimize thedistinctiveness of visual words. Large databases result in differentlysized descriptor clusters depending on the frequency of thecorresponding image textures (e.g. windows in an urban environment).Thus, using a fixed naive count to limit the quantization results issuboptimal. It is more reasonable to use the accumulated variance withrespect to the dimensions of all descriptors in a node as the criterionto stop the quantization. This approach reflects the true clusterstatistics, independent of the database size. However, the function usedto compute the descriptor vectors from image patches is highlynonlinear, which means that pairs of equally similar image patches donot result in equally similar descriptor distances and thus variances.Hence, a common threshold on the accumulated variance does not achieve afair quantization among the clusters.

The shape of the hyper-ellipsoid comprising the descriptors of a nodeindicates whether multiple distinct clusters are contained, which can beapproximated with the distribution of the variance among the dimensions.Hyper-ellipsoids with almost equal variance in the first principalcomponents (close to hyperspheres) can be assumed to contain only onedistinct cluster. In contrast to the accumulated variance, thisdistribution is not depending on the nonlinear mapping of thedescriptors. Further, it is not even necessary to compute this highdimensional variance distribution, as the separability and thus theshape of a node can be determined by the percentage of descriptors lyingin the overlapping buffer. If it is above a certain threshold, the shapeis close to a hypersphere, and the quantization process can be stoppedfor that node. Thus, an overfitting of descriptor clusters can beeffectively avoided resulting in smaller trees with significantlyincreased performance. With this modification, the tree is builtaccording to the definition of splitting axis described above. Based onthis indexing structure, the registration of the database images followsas described in Algorithm 1:

Algorithm 1 Register images For each image I  Initialize normalizationvector {right arrow over (n)} := {right arrow over (0)}  For eachfeature {right arrow over (f)}   Propagate {right arrow over (f)} downthe tree, including paths    reached because of spills   At each node iincrease inverted file entry    d_(i)(I) by IDF_(i)   At nodes i alongthe likeliest path increase    n_(i)(I) by IDF_(i)  Normalize {rightarrow over (d)}(I) by dividing by ∥{right arrow over (n)}(I)∥_(p)

At this point, it is assumed that the IDFs of every node have alreadybeen determined during the tree building process. Each featuredescriptor of every database image is propagated down the tree resultingin multiple paths and parallel inverted file entries if the descriptoris at least once assigned to an overlapping buffer. Instead of assigningall descriptors to multiple leaf nodes whose centroids are closer than acertain threshold, database descriptors are allowed to follow onlyhypothetical paths that a query descriptor could traverse. Thus, theprobability of finding the matching descriptor in the leaf the query hasbeen assigned to, is significantly increased while minimizing the lossin distinctiveness of the nodes.

To allow for a fair comparison between images of different descriptorcounts, the BOF vectors have to be normalized to length one. However,including the contribution of multiple hypothetical paths will increasethe norm relative to the properties of the tree, not necessarily to theproperties of the descriptors. This would also decrease BOF entriesresulting from descriptors lying far away from decision boundaries,which thus have not been assigned more than once. This is avoided bynormalizing the BOF vector only with respect to the most likely path.This is the path the descriptor would have followed without theoverlapping buffers.

As the distance between the query and database BOF vectors is calculatedin an inverted file approach, only those dimensions of the vectors arecompared, where both have nonzero entries and thus descriptors havescored. A query BOF vector thus compares always only to the entries inthe BOF vector of reference image j that correspond to a single validcombination of hypothetical descriptor paths. All other entries areignored as the corresponding entries in the query BOF vector are zero.Hence, querying the database with one of the referenced database imagesis guaranteed to result in the minimum distance.

The query process is illustrated in Algorithm 2:

Algorithm 2 Find matching images Initialize query BOF vector {rightarrow over (q)} := {right arrow over (0)} For each feature of the queryimage traverse down the tree  Increment TF_(i) for eyory feature atevery node i  and add node i to a list For every distinct node i in thelist calculate  q_(i) = TF_(i) · IDF_(i) Normalize {right arrow over(q)} using L_(p) norm For every distinct node i in list and every imageI  in node's inverted file increase distance Δ′(I) by  δ_(p) = |q_(i) −d_(i)(I)|^(p) − |q_(i)|^(p) − |d_(i)(I)|^(p), i.e.  δ₂ = −2 · q_(i) ·d_(i)(I) for L₂ distance and  δ₁ = |q_(i) − d_(i)(I)| − q_(i) − d_(i)(I)for L₁ distance. Distance Δ(I) is 2 + Δ′(I) for L_(p) norm

This algorithm can be further optimized for the L2 norm by partitioningthe scalar product. However, due to the optimizations described below,the computation is described generalized for any Lp norm. Multiplehypothetical paths are not assigned to the query descriptor since thiswould increase the query time.

Visual Word Weighting

As described above, the probability of assigning a descriptor to thecorrect child node depends on the distance d between the feature and theseparating hyperplane and the size of the overlapping buffer ob. Withthe aid of this buffer, the quantization effects are reduced. It isfurther desirable to account for the probability of assigning matchingquery and database descriptors to the same leaf node. To evaluate thestatistics of matching SURF descriptors, an exhaustive search among thefeatures of numerous image pairs has been performed and potentialmatches have been validated via spatial verification. Here, the baselinebetween the images was up to 11 m, which complies with the imageretrieval tasks described below.

The covariance matrix of the difference vectors between matching featuredescriptor pairs is illustrated in FIG. 6. It shows the strong variancedifferences among the dimensions caused by the spatial Gaussianweighting of the SURF descriptor. Further, only minor co-variancesbetween the dimensions can be observed emphasizing the good propertiesof the descriptor.

FIG. 7a shows the histogram of distance vectors, i.e., the difference oftwo matching descriptors, in one of the dimensions. The mean of thisdistribution is as expected zero, as it is the difference of twoidentically distributed random variables. The difference distributionalong all dimensions can be best described with the aid of a Laplaciandistribution (see FIG. 7a ).

Thus, the probability Pf that a matching feature is “incorrectly”quantized to the neighboring child node can be determined by thecumulative 64-dimensional Laplacian distribution function. Asillustrated in FIG. 7b , this probability corresponds to the integralover the gray area beyond the overlapping buffer (ob). Thus it dependson the distance d between the query feature and the separatinghyperplane.

It is now assumed that the distribution of the entries of the differencevector between a query and a matching reference descriptor areindependently distributed following a Laplacian distribution. Thus, onlythe one dimensional cumulative distribution needs to be considered todetermine the probability Pf of finding a match on the other side of thebuffer:

$\begin{matrix}{{{P_{f}(d)} = {\frac{1}{2}e^{\frac{{d + {ab}}}{\sigma}}}};\;{\sigma = \sqrt{\frac{{var}(D)}{2}}}} & {{Equation}\mspace{14mu}(5)}\end{matrix}$

Here, σ is relative to the variance of the matching descriptordifferences D.

The probability of assigning matching descriptors to the same visualword corresponds to the probability of quantizing matching features tothe same node (1-Pf) in all quantization steps m:α_(i)=Π_(m)(1=P _(fm))  Equation (6)

This probability αi can be utilized to weight the correspondingcomparisons between the query and the reference BOF vectors in thefollowing equation:

$\begin{matrix}{{\sum\limits_{i}^{\;}{\alpha_{i}{{q_{i} - d_{i}}}^{P}}} = {{\sum\limits_{{i❘d_{i}} = 0}^{\;}{\alpha_{i}{q_{i}}^{P}}} + {\sum\limits_{{i❘q_{i}} = 0}^{\;}{\alpha_{i}{d_{i}}^{P}}} + {\sum\limits_{i❘{q_{i} \neq {0\bigwedge d_{i}} \neq 0}}^{\;}{\alpha_{i}{{q_{i} - d_{i}}}^{P}}}}} & {{Equation}\mspace{14mu}(7)}\end{matrix}$

Thereby, the influence of unconfident visual word quantizations can bereduced. In this equation the comparison between a single query anddatabase image pair is shown iterating over all dimensions of the BOFvector. To keep the query time linear with the number of querydescriptors, an inverted file scheme is utilized to compute thedifference between the query and database BOF vectors. In this scheme,each leaf node references the BOF vector entries di of those databaseimages that quantized a descriptor in this leaf. While quantizing thedescriptors of the query image to determine its BOF vector entries, onlythe BOF entries of the images that were quantized to the same leaves canbe accessed. Thus, one is limited to those entries in the BOF vectorswhere a descriptor of both query and database image quantized into thesame leaf (qi≠0 and di≠0).

The sum in the above equation can be divided into three distinct parts.The third term can be easily computed as it is iterating only overdimensions where both a query and database descriptor are quantized intoone leaf. In the first two terms either the database image has no entry(di=0) or no query descriptor has traversed the corresponding node(qi=0).

These terms cannot be directly computed in an inverted file scheme. Thefirst term can be expressed by the subtraction between the overallweighted sum of qi and the sum of the weighted qi at di≠0, which bothcan be computed:

$\begin{matrix}{{\sum\limits_{{i❘d_{i}} = 0}^{\;}{\alpha_{i}{q_{i}}^{P}}} = {{\sum\limits_{i}^{\;}{\alpha_{i}{q_{i}}^{P}}} - {\sum\limits_{i❘{d_{i} \neq 0}}^{\;}{\alpha_{i}{q_{i}}^{P}}}}} & {{Equation}\mspace{14mu}(8)}\end{matrix}$

Since in the second term of Equation (8) no weights αi exist for qi=0,they are set to a constant value k (Equation (9)). Similarly to theprevious substitution, the second term is substituted with thesubtraction between the overall sum of di and the sum of di at qi≠0, asshown in Equation (10). Since the BOF vector q is normalized to lengthone, the norm is now k. Hence, only terms at qi≠0 and di≠0 remain inEquation (5), which can be computed within the inverted file approach.

$\begin{matrix}{\mspace{79mu}{{\sum\limits_{{i❘q_{i}} = 0}^{\;}{\alpha_{i}{d_{i}}^{P}}} = {\sum\limits_{{i❘q_{i}} = 0}^{\;}{k{d_{i}}^{P}}}}} & {{Equation}\mspace{14mu}(9)} \\{{\sum\limits_{{i❘q_{i}} = 0}^{\;}{k{d_{i}}^{P}}} = {{{\sum\limits_{i}^{\;}{k{d_{i}}^{P}}} - {\sum\limits_{i❘{q_{i} \neq 0}}^{\;}{k{d_{i}}^{P}}}} = {k - {k{\sum\limits_{i❘{q_{i} \neq 0}}^{\;}{d_{i}}^{P}}}}}} & {{Equation}\mspace{14mu}(10)}\end{matrix}$

The second term of Equation (7) actually accounts for those entries inthe BOF vector comparison where the query image does not provide adescriptor (qi=0) whereas the database image does. This is an importantinformation on the image pair similarity if regular images are comparedamong each other. However, it is possible to match regular mobile devicecamera frames to 360° panoramas. Thus, the video frame can only be asubsection of the panorama and at most the corresponding fraction can bematched. Hence, the constant k can be adjusted that weights this secondterm in Equation (7) accordingly. In this case k is the ratio betweenthe field of view of the camera image and the panorama. This novelweighting allows for effectively matching images that show completelydifferent portions of the scene and thus increases the performance ofthe algorithm significantly. Generally, this framework enablesintegrating prior knowledge into the image matching process withoutincreasing query time.

Geospatial Score Smoothing

Querying the panorama image database returns an ordered list ofpanoramas, or equivalently, as the panoramas are tagged with geographiccoordinates, a list of geographic locations. For each location in thelist, there is a score indicating how similar the respective location isto the query image. As adjacent locations usually share most visualfeatures, it can be assumed that a high score for one location justifiesa high score for adjacent locations, even if the image retrieval systemis unable to find some of the adjacent locations due to clutter,occlusions, and other factors. Therefore, the ordered list of panoramasis processed in a way such that a fraction f_(ij) of the score for eachpanorama P_(i) in the result list is applied to all panoramas P_(j) thatcould have been found within a predetermined radius around the panoramaP_(i). The fraction f_(ij) can be determined by taking the geospatialdistance between P_(i) and P_(j) into account, e.g., by using a Gaussianfunction of the distance. This way, close panoramas will be assigned alarge fraction of the score, while those farther away only a smallfraction. The overall score for a location is computed by adding up allweighted scores for that location (possibly including its own score ifthe location has been part of the query result).

The result of this processing is a new list of panorama images sorted bytheir respective similarity scores. Compared to the original result, theamount of false positives is greatly reduced as they usually are notsupported by scores from adjacent panoramas. At the same time, thenumber of correctly retrieved panoramas is increased, as clusters ofadjacent panoramas will take along nearby panoramas that otherwise wouldhave been missed.

Overlapping Sub-Trees

Preferably, a system for location recognition based on visual datashould not be bound to a confined area but allow for continuous andprecise position estimation at any place. However, the scalability ofstate-of-the-art visual location recognition systems is limited to fewsquare kilometers and performance decreases significantly once a certainlimit is reached. In conventional systems, often only the mostinformative features are selected allowing for an increase in the numberof database images by a factor of 10 resulting in approximately 2 km².However, covering a city like San Francisco with more than 100 km² isdifficult and may not necessary as prior knowledge on the locationusually exists. Prior information could for instance stem from themobile Cell-ID of the network provider enabling determination of aposition estimate in the range of a few hundred meters at most.Integrating this information into the location recognition processreduces the required resources with respect to memory as well as querytime and increases precision.

Spatial Partitioning

According to an embodiment of the present invention, the search area issegmented, i.e., a large environment like a city, into severaloverlapping sub-regions. For each of them an individual tree andassociated inverted file set is generated. The size of the sub-regionscan be set to approximately 2 km² depending on the expected priorlocation information. In the example in FIG. 8, the area in the centeris surrounded by eight neighboring regions. Each of them is overlappingwith horizontally and vertically adjacent regions by 50%. Thus, everydistinct location is covered by four tree structures. At larger searchspaces, the pattern is continued accordingly.

In the localization process only those trees are queried that compriselocations within the area of the current uncertainty about the positionof the user. This results in four redundant similarity estimates perdatabase image for one query image. Thus, the computational complexityefficiently adapts to the current prior location information. Sincethese similarity estimates are determined by trees built with asignificantly differing image set, the quantization structure of each ofthem is slightly different from the others. The similarity scores areyet comparable and can be fused, as described below. This is similar toknown techniques like Locality Sensitive Hashing or Extremely RandomizedForest, where the results of a large number of random quantizationstructures are fused. However, in contrast to these techniques, the treeis not randomized directly. Instead optimal quantization structures arecomputed for different image sets. Thus, combining the redundant resultssignificantly increases the precision at the cost of increasing thequery time. While scaling the computational complexity with the locationuncertainty and increasing the precision, this system further enables aseamless handover to adjacent regions. Once a user is approaching theborder of the currently covered search space, i.e., the area covered byfour trees per sub-region, adjacent trees are preloaded. Thus, theposition of the user is always close to the center of one of the trees.

Score Fusion

The fusion of the individual sub-tree results can be based not only onthe returned similarity scores themselves but on the spatialrelationship of the retrieved locations.

The main steps of the fusion are outlined in the following algorithm:

Algorithm 3 Fuse subtree similarity estimates Select top K imagesretrieved from every queried subtree For every location that isretrieved from at least 2 trees  Calculate mean of the 2 most similarscores  Divide the score by the number of trees voting for this -  oradjacent locations Determine the top K locations with the lowest finalscore

In a first step, the top K results of every queried sub-tree areselected. This number is determined by optimizing the tradeoff betweenretrieval precision and recall. While it may be preferable to retrieveideally only images in the surrounding of the query (precision), it mayalso be desirable to retrieve as many images in the surrounding aspossible (recall). Typically, K is in the range of 10 to 20 images.Subsequently, the locations that have been retrieved from at least twotrees are selected to reduce the amount of unreliable matches, using theredundancy provided by the four independent quantization structures. Forthese locations, the mean of the similarity measures is computed basedon the two most similar measures to increase robustness againstoutliers.

As it is likely that, due to clutter and dynamic objects in thedatabase, not the same but adjacent locations are retrieved, theinformation implied in local clusters of retrieved images is of centralimportance. Thus, the mean similarity score of every distinct locationis divided by the number of trees voting for this or adjacent locations.Finally, at most K locations sorted with respect to the final score arereturned. Hence, this algorithm effectively increases the precision ofthe returned results by fusing the information of multiple trees at verylow computational complexity. Based on these results a post-processingcan be performed via geometric consistency verification. The knowledgeon the spatial relation between database images can be furtherintegrated with the aid of temporal Bayesian filtering.

Location Recognition

FIG. 9 illustrates a method of location recognition according to anembodiment of the invention. First, Hessian Affine interest points areextracted from 12 gnomonic projections of every panorama and modifiedSURF descriptors are computed. The search area is subdivided into 3 by 3overlapping sub-regions, each covering 25% of the overall area. Theindividual trees are trained with the corresponding sub-region datasetand conservative quantization parameters.

As query images, video frames may be recorded by a handheld device, forexample at a resolution of 640 by 480 pixels and 6 fps. To reduce theamount of clutter features queried, an efficient tracking is performedby matching the Hessian Affine operator responses instead of descriptorvectors. Only those interest points are considered that can be trackedover multiple frames, which reduces the amount of features caused byocclusions or being susceptible to affine transformations. The query iscomposed of all accepted features visible within the last five secondsassuming pedestrian motion. That way, small virtual feature panoramasare generated, caused by the usual camera pans.

The feature set related to the current frame is queried in all treescorresponding to a sub-region that overlaps with the current positionuncertainty. Thus, regular images are matched against panoramas,allowing for an arbitrary viewing direction.

Incorrect results may be caused by locations with similar appearance.However, fusing the information of the four queried trees, as describedabove, effectively excludes almost all false matches. This informationintegration is even more effective if less optimal sub-tree resultsoccur.

A triangulation via neighboring panoramas may be carried out to performa final validation as well as precise location and orientationdetermination. Missed localizations, which occur if the query featureset is degenerated, may be overcome with the aid of probabilisticfiltering and visual odometry.

Additional Embodiments

In one embodiment there is provided a visual localization methodcomprising: providing access to a database that is maintained by:electronically storing a plurality of images in a database, wherein eachof the stored images is associated with a location, and wherein each ofthe stored images is represented by a reference vector indicative of thefrequency of visual words in the respective image; determiningdescriptors corresponding to visual features in the plurality of storedimages; generating a binary search tree by quantizing the descriptors toobtain a plurality of visual words corresponding to nodes of the binarysearch tree, comprising: determining a hyperplane for each node toseparate the child nodes of the respective node; and storing descriptorsthat are within a distance from the hyperplane of a node in anoverlapping buffer associated with that node; for a given node,terminating the quantization when reaching a predetermined quantizationthreshold; the visual location method further comprising: receiving oneor more video frames representing a query image of the location to berecognised; determining descriptors corresponding to visual features inthe query image; quantizing said descriptors to obtain a plurality ofvisual words; and using an inverted file index to determine an image inthe database that contains said visual words and the location associatedwith that image.

Preferably the predetermined quantization threshold is reached whenthere is only a single cluster of descriptors in the given node, or thenumber of descriptors in the given node reaches a threshold, or apredetermined portion of descriptors has been stored in the overlappingbuffer associated with the given node.

Preferably determining the hyperplane for a given node comprisesdetermining a vector spanning between the two farthest descriptorcluster centres, preferably with round (√{square root over (n)}·c)descriptors per cluster, wherein n is the number of descriptors in thegiven node and c is a selectable coefficient preferably set to around25%.

Preferably in the database an inverted file index is generated for eachleaf node of the binary search tree to reference images in the databasethat contain visual features corresponding to descriptors quantized inthe respective leaf node.

The method may also comprise a segmenting the region in which the visuallocation recognition is to be performed into a plurality of overlappingsub-regions.

The method may further comprise generating a respective sub-tree foreach of the sub-regions, thereby providing a plurality of overlappingbinary trees; and generating a respective inverted file index for eachof the sub-regions.

The method may further comprise performing an approximation of thelocation of a user providing the query image; identifying one or moresub-regions based on the approximation; and performing a visual locationrecognition method of any of any preceding claim in the identifiedsub-region(s) only.

The method may further comprise performing a visual location recognitionmethod of any of claims 1 to 6 in two or more of the sub-regions toobtain respective similarity measures; and fusing the similaritymeasures to determine a location.

The method may further comprise pre-selecting a predetermined number ofthe best similarity measures of every queried binary sub-tree; from thepre-selected similarity measures, selecting the similarity measurescorresponding to locations that have been obtained from at least twosub-trees; for the selected similarity measures, determining the mean ofthe similarity measures based on the two or more most similar measures;and using the number of sub-trees pointing to an adjacent location tocompute an overall similarity measure of every distinct location.

Preferably said distance is determined in accordance with a distributionof descriptors in the associated node.

The method may further comprise determining the distance between a queryvector representative of the frequency of visual words in the queryimage and said reference vectors, wherein preferably only dimensions ofthe vectors are compared that both have nonzero entries.

The method may further comprise weighting the contribution of visualwords to the difference between said query vector and said referencevector.

Preferably the weighting is performed in accordance with the probabilitythat a query descriptor is allocated to the same visual word as acorresponding descriptor in the database, and/or in accordance with thereliability of a query descriptor.

The method may further comprise using the inverted file index to computethe distance between a query vector and said reference vectors.

In an embodiment there is provided an article comprising a mediumstoring instructions that, when executed, cause a processor-based systemto perform any of the steps of the above described methods.

It will be appreciated that the above described embodiments aredescribed as examples only, and that modifications to these embodimentsare included within the scope of the appended claims.

The invention claimed is:
 1. A visual object search method comprising:(a) maintaining, by a server, reference data in a database providing adistinctive description of recorded objects; (b) receiving, by theserver, information about the context of a user device; (c) determining,by the server based on the received data, a subset of the referencedata; (d) transmitting the subset of the reference data from the serverto the user device; (e) storing, by the user device, the subset of thereference data; (f) generating, by the user device, visual feature datarepresentative of one or more visual features detected by the userdevice; (g) retrieving, by the user device, an object based on at leastthe visual feature data and the subset of the reference data; (h)repeating at least steps (f) and (g) at least twice every time thesubset of the reference data is obtained in step (d).
 2. The method ofclaim 1, wherein said visual feature data include information on itscontent.
 3. The method of claim 2, wherein said content includes atleast one trademark.
 4. The method of claim 1, wherein said object is aproduct.
 5. The method of claim 2, wherein said object is a product andsaid content is the trademark of said product.
 6. A mobile visual objectsearch method comprising: (a) maintaining, by a server, reference datain a georeferenced database providing a distinctive description ofrecorded objects; (b) receiving, by the server, information about theenvironment of a mobile device; (c) determining, by the server based onthe received data, a subset of the reference data; (d) transmitting thesubset of the reference data from the server to the mobile device; (e)storing, by the mobile device, the subset of the reference data; (f)generating, by the mobile device, visual feature data representative ofone or more visual features detected by the mobile device; (g)retrieving, by the mobile device, an object based on at least the visualfeature data and the subset of the reference data; (h) repeating atleast steps (f) and (g) at least twice every time the subset of thereference data is obtained in step (d).
 7. The method of claim 6,wherein said visual feature data in step (f) include information on itscontent.
 8. The method of claim 7, wherein said content includes atleast one exhibit.
 9. The method of claim 7, wherein said contentincludes at least one store name.
 10. The method of claim 7, whereinsaid content includes at least one trademark.
 11. The method of claim 6,wherein said object includes at least one product.
 12. The method ofclaim 10, wherein said object is a product and said trademark is thetrademark of said product.
 13. The method of claim 6, wherein theinformation about the environment of the mobile device is derived fromthe approximate position of the mobile device.